Field of the Invention
This invention relates to the fields of microscopic imaging of large specimens with particular emphasis on brightfield and fluorescence imaging, including photoluminescence and spectrally-resolved fluorescence. Applications include imaging tissue specimens, genetic microarrays, protein arrays, tissue arrays, cells and cell populations, biochips, arrays of biomolecules, detection of nanoparticles, photoluminescence imaging of semiconductor materials and devices, and many others.
Description of the Prior Art
The macroscope originally described in U.S. Pat. No. 5,381,224 is a scanning-laser system that uses a telecentric laser-scan lens to provide a wide field of view. Several embodiments are presently in use. These include instruments for fluorescence and photoluminescence (including spectrally-resolved) imaging (several other contrast mechanisms are also possible), instruments in which a raster scan is provided by the combination of a scanning mirror and a scanning specimen stage, instruments in which the specimen stage is stationary and the raster scan is provided by two scanning mirrors rotating about perpendicular axes, confocal and non-confocal versions, and other embodiments. A macroscope with fine focus adjustment was described in U.S. Pat. No. 7,218,446, and versions for reflected-light, fluorescence, photoluminescence, multi-photon fluorescence, transmitted-light, and brightfield imaging were described. The combination of a scanning laser macroscope with a scanning laser microscope to provide an imaging system with a wide field of view and the high resolution capability of a microscope was described in U.S. Pat. No. 5,532,873.
When the macroscope is used for fluorescence imaging, it has several advantages. Exposure for each fluorophore can be adjusted separately without changing scan speed by changing either laser intensity and/or detector gain (in the case of a detector comprised of a photomultiplier tube (pmt) followed by a preamplifier, both the pmt voltage (which changes pmt gain) and preamplifier gain can be changed). The ability to adjust the detection gain for each fluorophore separately allows the instrument to simultaneously collect multiple fluorophore images that are all correctly exposed. In addition, the appropriate laser wavelength can be provided to excite a chosen fluorophore, and excitation wavelengths can be chosen so they do not overlap detection wavelength ranges.
Several other technologies are used for imaging large specimens at high resolution. With tiling microscopes, the image of a small area of the specimen is recorded with a digital camera (usually a CCD camera), the specimen is moved with a computer-controlled microscope stage to image an adjacent area, an image of the adjacent area is recorded, the stage is moved again to the next area, and so on until a number of image tiles have been recorded that together cover the whole area of the specimen. Images of each area (image tiles) are recorded when the stage is stationary, after waiting long enough for vibrations from the moving stage to dissipate, and using an exposure time that is sufficient to record the fluorescence images. These image tiles can be butted together, or overlapped and stitched using computer stitching algorithms, to form one image of the entire specimen. Such images may contain tiling artifacts, caused by focus changes between adjacent tiles, differences in illumination intensity across the field of view of the microscope, barrel or pincushion distortion near the edge of the tiles, and microscope objectives that do not have a flat focal plane. For large specimens, thousands of tiles may be required to image the entire specimen, increasing the chance of tiling artifacts. Tiling microscopes are very slow for fluorescence imaging.
When tiling microscopes are used for fluorescence imaging, the areas surrounding each tile and the overlapping edges of adjacent tiles are exposed twice (and the corners four times) which can bleach some fluorophores. Exposure is adjusted by changing the exposure time for each tile. If multiple fluorophores are imaged, a different exposure time is required for each, so each fluorophore requires a separate image at each tile position. Multiple exposure of the specimen for imaging multiple fluorophores can also increase bleaching. After all tiles have been collected, considerable effort (both human and computer) is required to stitch the tiles together and correct each tile for illumination intensity and collection sensitivity changes across the field of view of the microscope (correction for variations in illumination intensity and collection sensitivity is sometimes called “field flattening”). Stitching tiles together is also complicated by distortion and curvature of field of the microscope objective, which occur near the edges of the field of view (just where stitching of tiles occurs).
Strip scanning instruments are also used for imaging large specimens. In these instruments infinity-corrected microscope optics are used, with a high Numerical Aperture (high NA) microscope objective and a tube lens of the appropriate focal length to focus an image of the specimen directly on a CCD or CMOS linear array sensor or TDI sensor with the correct magnification to match the resolution of the microscope objective with the detector pixel size for maximum magnification in the digitized image {as described in “Choosing Objective Lenses: The Importance of Numerical Aperture and Magnification in Digital Optical Microscopy”, David W. Piston, Biol. Bull. 195, 1-4 (1998)}. A linear CCD detector array with 1000 or 2000 pixels is often used, and three separate linear detectors with appropriate filters to pass red, green and blue light are used for RGB brightfield imaging. The sample is moved at constant speed in the direction perpendicular to the long dimension of the linear detector array to scan a narrow strip across a microscope slide. The entire slide can be imaged by imaging repeated strips and butting them together to create the final image. Another version of this technology uses linear TDI (Time Delay Integration) array sensors which increase both sensitivity and imaging speed. In both of these instruments, exposure is varied by changing illumination intensity and/or scan speed.
Such a microscope is shown in FIG. 1 (Prior Art). A tissue specimen 100 (or other specimen to be imaged) mounted on microscope slide 101 is illuminated from below by illumination source 110. Light passing through the specimen is collected by infinity-corrected microscope objective 115, which is focused on the specimen by piezo positioner 120. The microscope objective 115 and tube lens 125 form a real image of the specimen on linear detector array 130. An image of the specimen is collected by moving the microscope slide at constant speed using motorized stage 105 in a direction perpendicular to the long dimension of the detector array 130, combining a sequence of equally-spaced line images from the array to construct an image of one strip across the specimen. Strips are then assembled to form a complete image of the specimen.
For brightfield imaging, most strip-scanning instruments illuminate the specimen from below, and detect the image in transmission using a sensor placed above the specimen. In brightfield, signal strength is high, and red, green and blue channels are often detected simultaneously with separate linear detector arrays to produce a colour image.
Compared to brightfield imaging, fluorescence signals can be thousands of times weaker, and some fluorophores have much weaker emission than others. Fluorescence microscopy is usually performed using illumination from the same side as detection (epifluorescence) so that the bright illumination light passing through the specimen does not enter the detector. In strip-scanning instruments, exposure is varied by changing scan speed, so present strip-scanning instruments scan each fluorophore separately, reducing the scan speed when greater exposure is required for a weak fluorophore. Varying exposure by changing scan speed makes it difficult to design a strip-scanner for simultaneous imaging of multiple fluorophores, where each channel would have the same exposure time, and present strip-scanners scan one fluorophore at-a-time. In addition, in fluorescence microscopy, relative intensity measurements are sometimes important for quantitative measurement, and 12 or 16 bit dynamic range may be required. For present strip scanners, this would require larger dynamic range detectors and slower scan speeds.
Before scanning a large specimen in fluorescence, it is important to set the exposure time (in a tiling or strip-scanning microscope) or the combination of laser intensity, detector gain and scan speed (in a scanning laser macroscope or microscope) so that the final image will be properly exposed—in general it should not contain saturated pixels, but the gain should be high enough that the full dynamic range will be used for each fluorophore in the final image. Two problems must be solved to achieve this result—the exposure must be estimated in advance for each fluorophore and for simultaneous detection of multiple fluorophores, the exposure time must be adjusted separately for each detection channel before scanning. For strip-scanning instruments, estimating the exposure in advance is difficult without scanning the whole specimen first to check exposure, and this must be done for each fluorophore. Instead of scanning first to set exposure, many operators simply set the scan speed to underexpose slightly, with resulting noisy images, or possibly images with some overexposed (saturated) areas if the estimated exposure was not correct. For macroscope-based instruments, a high-speed preview scan can be used to set detection gain in each channel before final simultaneous imaging of multiple fluorophores (see WO2009/137935, “Imaging System with Dynamic Range Maximization”).
A prior art scanning microscope for fluorescence imaging is shown in FIG. 2. A tissue specimen 100 (or other specimen to be imaged) mounted on microscope slide 101 is illuminated from above by illumination source 200. In fluorescence imaging the illumination source is usually mounted above the specimen (epifluorescence) so that the intense illumination light that passes through the specimen is not mixed with the weaker fluorescence emission from the specimen, as it would be if the illumination source were below the specimen. Several different optical combinations can be used for epifluorescence illumination—including illumination light that is injected into the microscope tube between the microscope objective and the tube lens, using a dichroic beamsplitter to reflect it down through the microscope objective and onto the specimen. In addition, a narrow wavelength band for the illumination light is chosen to match the absorption peak of the fluorophore in use. Fluorescence emitted by the specimen is collected by the infinity-corrected microscope objective 115 which is focused on the specimen by the piezo positioner 120. Emission filter 205 is chosen to reject light at the illumination wavelength and to pass the emission band of the fluorophore in use. The microscope objective 115 and the tube lens 125 form a real image of the specimen on TDI detector array 210. An image of the specimen is collected by moving the microscope slide at constant speed using the motorized stage 105 in a direction perpendicular to the long dimension of the detector array 210, combining a sequence of equally-spaced, time-integrated line images from the array to construct an image of one strip across the specimen. Strips are then assembled to form a complete image of the specimen. When a CCD-based TDI array is used, each line image stored in memory is the result of integrating the charge generated in all of the previous lines of the array while the scan proceeds, and thus has both increased signal/noise and amplitude (due to increased exposure time) when compared to the result from a linear array detector. Exposure is also increased by reducing scan speed, so the scan time (and thus image acquisition time) is increased when using weak fluorophores. In addition, it is difficult to predict the best exposure time before scanning. When multiple fluorophores are used on the same specimen, the usual imaging method is to choose illumination wavelengths to match one fluorophore, select the appropriate emission filter and scan time (speed) for the chosen fluorophore, and scan one strip in the image. Then, the illumination wavelength band is adjusted to match the absorption band of the second fluorophore, a matching emission filter and scan speed are chosen, and that strip is scanned again. Additional fluorophores require the same steps to be repeated. Finally, this is repeated for all strips in the final image. Some instruments use multiple TDI detector arrays to expose and scan multiple fluorophores simultaneously, but this usually results in a final image where one fluorophore is exposed correctly and the others are either under- or over-exposed. Exposure can be adjusted by changing the relative intensity of the excitation illumination for each fluorophore, which should be easy to do if LED illumination is used. When multiple illumination bands are used at the same time, the resulting image for each fluorophore may differ from that produced when only one illumination band is used at a time because of overlap of the fluorophore emission bands, and because autofluorescence from the tissue itself may be excited by one of the illumination bands. Autofluorescence emission usually covers a wide spectrum and may cause a bright background in all of the images when multiple fluorophores are illuminated and imaged simultaneously.
A good description of strip scanning instruments, using either linear arrays or TDI arrays, is given in US Patent Application Publication No. US2009/0141126 (“Fully Automatic Rapid Microscope Slide Scanner”, by Dirk Soenksen).
Linear arrays work well for brightfield imaging, but the user is often required to perform a focus measurement at several places on the specimen before scanning, or a separate detector is used for automatic focus. Linear arrays are not often used for fluorescence imaging because exposure time is inversely proportional to scan speed, which makes the scan time very long for weak fluorophores. In addition, exposure (scan speed) must be adjusted for each fluorophore, making simultaneous measurement of multiple fluorophores difficult when they have widely different fluorescence intensity (which is common).
TDI arrays and associated electronics are expensive, but the on-chip integration of several exposures of the same line on the specimen provides the increased exposure time required for fluorescence imaging while maintaining a reasonable scan speed. Simultaneous imaging of multiple fluorophores using multiple TDI detector arrays is still very difficult however, since each of the detectors has the same integration time (set by the scan speed), so it is common to use only one TDI array, adjusting exposure for each fluorophore by changing the scan speed and collecting a separate image for each fluorophore. Focus is set before scanning at several positions on the specimen, or automatic focus is achieved using a separate detector or focus measuring device.
Single-chip colour cameras (including those often used on ordinary optical microscopes to record an image of the specimen area seen in the field of view of the microscope) often use a mosaic Colour Filter Array (CFA) on the photosensors of an area detector that enables a single-chip camera to record a colour image. The most common CFA is the Bayer filter, named after the inventor, which arranges red, green and blue filters in a square grid (usually RGGB) of photosensors on the array (see U.S. Pat. No. 3,971,065). Bayer chose to use two green pixels in each set as luminance elements to match the sensitivity of the human eye. Data from each pixel only records one colour, so in order to obtain a full-colour image, the red, green and blue values for each pixel are calculated by interpolation using information from the surrounding pixels. This is called demosaicing, and several different demosaicing algorithms have been developed. Demosaicing may happen inside the camera (producing jpeg or tiff files), or outside using raw data from the sensor. Because of the computing power required for demosiacing, and the possibility of false colours and moiré, most tiling microscopes use separate array detectors to detect red, green and blue. Cameras for digital photography often use an optical low-pass filter in front of the detector array to reduce moiré and false colour caused by use of a Bayer filter, resulting in some loss of resolution. Other mosaic Colour Filter Arrays have been proposed, including one using a white (transparent) filter in place of one of the green filters in the Bayer array. White (transparent) pixels accept all wavelengths of light in the visible spectrum (they are panchromatic), and provide one bright pixel in each array of four pixels, increasing the sensitivity of the array. Other combinations of filters in the square mosaic grid include CYGM (cyan, yellow, green, magenta) and RGBE (red, green, blue, emerald). All require demosaicing.